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Research-creation project to develop hardware and software 
for use in live performance (theater, music, dance) and 
interactive environments. 


Components: 

■ small, battery powered wireless PCB’s for sensing and 
actuation, 

■ software for real-time sharing of data, and 

■ modules to analyse datastreams and generate complex 
dynamics for output media. 
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■ Economic and technical constraints of live performance 


■ Lack of tools for artistic use 


■ Real world testing scenarios 
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■ Economic and technical constraints of live performance 
Limited time in rehearsals to use and test technology 

■ Lack of tools for artistic use 

Low cost, integration with common software environments 

■ Real world testing scenarios 

Application in professional productions, outside of the lab 
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■ Low cost 

■ Small form factor 

■ Flexible sensor configuration 

■ Usable for control of motors, LEDs, and other actuators 

■ Operable in large groups (10+ nodes) 

■ Long battery life 

■ Ease of use 

■ Programmable, so that the board can take care of more logic 
and processing of data, if desired by the user 
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■ Small size: 42mm x 35mm 

■ Based on Arduino Mini Pro 

■ Useable with XBee radio (mesh network) 

■ Unit cost price 32 CAD (excluding XBee) 
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8 analog inputs (left side): A4 (SDA), A5 (SCL), AO, A1, A2, A3, A6, A7 
11 digital inputs or outputs (right side): D3 to D13 
Serial I/O (top left): RX, TX 

Power input (between 3.3V and 5V) (left side bottom): RAW, GND 
Regulated power output (left side bottom inside): 3.3V, GND 
I2C communication, (left side top): SDA, SCL 
PWM output, (right side), D3, D5, D6, D9, DIO, D11 
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Firmware library supports: 


■ Analog sensors (connected to the analog input pins, e.g. 
resistive sensors, analog accelerometers, infrared distance 
sensors) 

■ Digital sensors (on/off, e.g. buttons and switches) 

■ LIS302DL accelerometer, using I2C 

■ Relative humidity and temperature sensor 

■ Ultrasound sensors 

■ PWM output (e.g. dimmable LEDs, motors) 

■ Digital output (on/off) 
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Wireless configuration of the firmware (compare to Firmata) 


■ Board reads the serial number of the XBee and sends it to 
the coordinator 

■ Coordinator assigns an ID and tells the board whether or not 
it will receive a new configuration 

♦ Board sends a message back that it is waiting 

♦ Coordinator sends configuration 

■ Board sends a summary of the current configuration (for the 
coordinator to verify) 

■ Board starts sending data 
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Wireless configuration of the firmware (compare to Firmata) 


■ Board reads the serial number of the XBee and sends it to 
the coordinator 

■ Coordinator assigns an ID and tells the board whether or not 
it will receive a new configuration 

♦ Board sends a message back that it is waiting 

♦ Coordinator sends configuration 

■ Board sends a summary of the current configuration (for the 
coordinator to verify) 

■ Board starts sending data 

Firmware can be customized for tasks not implemented in our 

library. 
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The aim of the SenseWorld DataNetwork is to facilitate 

■ sharing and 

■ manipulation of 

■ multiple data streams 

■ between collaborators 

■ in heterogeneous interactive performance environments 
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The aim of the SenseWorld DataNetwork is to facilitate 

■ sharing and 

■ manipulation of 

■ multiple data streams 

■ between collaborators 

■ in heterogeneous interactive performance environments 

Personal experience in interactive dance/theatre: sharing data 
between light, video, sound and sensing 
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The aim of the SenseWorld DataNetwork is to facilitate 

■ sharing and 

■ manipulation of 

■ multiple data streams 

■ between collaborators 

■ in heterogeneous interactive performance environments 

Personal experience in interactive dance/theatre: sharing data 
between light, video, sound and sensing 


Other frameworks: 

■ Mapper of the Digital Orchestra project 

■ KeyWorx 
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■ OpenSoundControl provides a low level communication 
protocol 

■ and is used by many programs for interactive media (e.g. 
Supercollider, Max/MSP, PureData, Processing,...) 

■ But there is no higher level protocol to facilitate the 
communication 
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■ OpenSoundControl provides a low level communication 
protocol 

■ and is used by many programs for interactive media (e.g. 
Supercollider, Max/MSP, PureData, Processing,...) 

■ But there is no higher level protocol to facilitate the 
communication 


Support coordinated collaboration with real time data within a 
live performance context, involving multiple types of media 


LAC - May 1-4, 2010 


Sense/Stage - wireless sensor infrastructure for realtime interactive environments - p. 12/33 












Concept 


» Overview 

Introduction and motivation 


SenseStage MiniBee 


SenseWorld DataNetwork 


» SenseWorld DataNetwork 
» Motivation 


» Concept 


» Design criteria 
» Client connection 
» Auto-recovery 

Clients 


Integration 


Examples 


Conclusion 



LAC - May 1-4, 2010 


Sense/Stage - wireless sensor infrastructure for realtime interactive environments - p. 13/33 























Design criteria 


» Overview 

Introduction and motivation 


SenseStage MiniBee 


SenseWorld DataNetwork 


» SenseWorld DataNetwork 
» Motivation 
» Concept 


» Design criteria 


» Client connection 
» Auto-recovery 

Clients 


Integration 


Examples 


Conclusion 


■ Any client should be able to subscribe to data 

■ Any client should be able to supply data 

■ Restore configuration quickly 

■ Usable within heterogeneous media software environments 

■ Enable collaboration between different design practices 

■ Enable efficiency of collaboration within the limited 
timeframe of rehearsals 

■ Integration with the wireless sensor network 
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■ Any client should be able to subscribe to data 

■ Any client should be able to supply data 

■ Restore configuration quickly 

■ Usable within heterogeneous media software environments 

■ Enable collaboration between different design practices 

■ Enable efficiency of collaboration within the limited 
timeframe of rehearsals 

■ Integration with the wireless sensor network 

The framework defines an OSC-namespace for communication 

between host and clients, 
for exchanging and labeling data 
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The framework defines an OSC-namespace for communication 

between host and clients, 
for exchanging and labeling data 


DataNetwork the network itself 

DataNode a node is a collection of slots, usually based upon 
a device or another common source 

DataSlot a slot is a single data stream 

Client client program receiving and/or setting data 
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■ The host writes a textfile with the network’s OSC port can be 
found in the file 

http://192.168.1.7/SenseWorldDataNetwork 

■ The host broadcasts an announce message 
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■ The host writes a textfile with the network’s OSC port can be 
found in the file 

http://192.168.1.7/SenseWorldDataNetwork 

■ The host broadcasts an announce message 

■ The client sends a register message 

■ The client has to reply to regularly sent ping messages to 
confirm its presence 

■ The client queries which nodes and slots are present 
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■ The host writes a textfile with the network’s OSC port can be 
found in the file 

http://192.168.1.7/SenseWorldDataNetwork 

■ The host broadcasts an announce message 

■ The client sends a register message 

■ The client has to reply to regularly sent ping messages to 
confirm its presence 

■ The client queries which nodes and slots are present 

■ The client subscribes to nodes/slots of interest 

■ The client can set data to nodes 

■ The client is informed of new or changed nodes that are 
created after registration 
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Sometimes software and/or hardware is unexpectedly and 
fatally interrupted, even during rehearsals and performances. 

■ The host writes a textfile with the network’s OSC port can be 
found in the file 

http://192.168.1.7/SenseWorldDataNetwork. 

-> The client looks this file up, upon startup 


LAC-May 1-4, 2010 


Sense/Stage - wireless sensor infrastructure for realtime interactive environments - p. 16/33 















Auto-recoverv 


r 


» Overview 

Introduction and motivation 


SenseStage MiniBee 


SenseWorld DataNetwork 


» SenseWorld DataNetwork 
» Motivation 
» Concept 
» Design criteria 
» Client connection 


» Auto-recovery 


Clients 


Integration 


Examples 


Conclusion 


Sometimes software and/or hardware is unexpectedly and 
fatally interrupted, even during rehearsals and performances. 

■ The host writes a textfile with the network’s OSC port can be 
found in the file 

http://192.168.1.7/SenseWorldDataNetwork. 

-> The client looks this file up, upon startup 

■ The host broadcasts an announce message 
-> The client reacts with register message 
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Host: 

x = SWDataNetwork.new; // define the network 
x.createHost; 


Client: 

Y = 

SWDataNetworkClient.new("192.168.0.104","Marije"); 
y.makeGui; 

y.addExpected( 100, "testnode", 5 ); 
y.setData( 100, [0,1,2,3,4] ); 

y.queryNodes; // query available nodes 
y.subscribeNode( 101 ); // subscribe to node 101 
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import datanetwork.*; 

DNConnection dn; //DNConnection instance 
DNNode node; //DNNode instance 

void setup() { 

dn = new DNConnection(this, "192.168.0. 

dn.getServerPort("192.168.0.104’ 
6009, "p5Client"); 

node = new DNNode(2000, 5, 0, "p5Node") 

} 

void stop () { 

dn.unsubscribeAll(); 
dn.removeAll(); 
dn.c1o s e (); 


10 4", 

) , 
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void keyPressed() 
if (key == 'r') 
else if (key == 
else if (key == 
else if (key == 
new float[] 


{ 

dn.register(); 

'q') dn.queryAll(); 

'f') dn.subscribeNode(401) 
' d') dn.setData(node, 

{ 4.0, 2.0, 1.0, 2.3, 4.4 } 


} 


r 


void dnEvent(String addr, float[] args) { 
print ( "Float: M + addr); 
for(int i = 0;i < args.length;i++) 
print (" M +args[i]); 
printIn(); 
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DataNode * node; 

DataNetwork * dn; 

dn = new DataNetwork(); 
dn->createOSC( "127.0.0.1", 7000, 
dn->registerMe(); 
dn->query(); 

dn->subscribeNode( 5, true ); 

dn->createNode( 4, "world", 5, 0, 

float dummydata[] = {0.1, 0.3, 0. 

node = dn->getNode( 4 ) ; 
node->setData( 5, dummydata); 
node->send( true ); 


"libdn" ); 

true ); 

4, 0.5, 0.6}; 
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■ dn.node - get data from node(s) 

■ dn.makenode - supply data to a node 

■ dn->dot - bridge from the datanetwork to the digital 
orchestra mapping tools 
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dn.node 


Easy interaction with the SenseStage Data Network! 


pv dn-host 192.168.0.104 


dn.node 114 

T- 


114 149 127 124 120 


Arguments specify nodes to subscnbe to. 
Bang in inlet re-initializes. as does "init" 
message sent to "dn.node". 

i 


dn.node 223 224 

T- 


224 0.521569 0.937255 0.909804 0.815686 
0.917647 0.85098 0.964706 0.862745 


Examples 


Conclusion 


init 


.1 

s dn.node 


This abstraction learns the host IP address 
using the "pv dn-host” object - this MUST be 
included in the parent patcher. 


Digital 
Orchestra 
Toolbox 

www.idmil.org 

[ (dot 3dpolar 
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« n o 


o dn.makenode.maxhelp 


dn.makenode 


Easy interaction with the SenseStage Data Network! 


pv dn-host 192.168.0.104 


Arguments: Node, Number of Slots, 
and Name of the Node 


I-,-. 

0.1 3.4 9 

r r r 

pack 0. 0. 0. 

7 

dn.makenode 999 3 TestData 


Send data as a list of floats. Each 
element corresponds to a slot. 


Bang in right inlet 
re-initializes, as does 
"Init" message sent to 
"dn.node". 


init 

1 


s dn.node 


dn.node 999 

T 


999 0.1 3.4 9. 


This abstraction learns the host IP address 
using the "pv dn-host" object - this MUST be 
included in the parent patcher. 
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■ Reading in data from the wireless nodes in the DataNetwork 
host 

■ Each MiniBee “node” corresponds to a DataNode 

■ Each sensor on the MiniBee corresponds to a DataSlot 
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Conclusion 


■ Reading in data from the wireless nodes in the DataNetwork 
host 

■ Each MiniBee “node” corresponds to a DataNode 

■ Each sensor on the MiniBee corresponds to a DataSlot 

■ Dialog for wireless configuration of nodes 

■ Storing current configuration of all nodes with DataNetwork 
setup 


LAC - May 1-4, 2010 


Sense/Stage - wireless sensor infrastructure for realtime interactive environments - p. 25/33 












» Overview 

Introduction and motivation 
SenseStage MiniBee 
SenseWorld DataNetwork 
Clients 


Integration 


Examples 


» Chronotopia 
» JND/Semblance 
» SenseStage Workshop 

Conclusion 


Examples 


LAC - May 1-4, 2010 


Sense/Stage - wireless sensor infrastructure for realtime interactive environments - p. 26/33 










Chronotopia 


» Overview 

Introduction and motivation 
SenseStage MiniBee 
SenseWorld DataNetwork 
Clients 


Integration 


Examples 
» Chronotopia 


» JND/Semblance 
» SenseStage Workshop 

Conclusion 


Chronotopia - a dance piece with the Attakkalari Centre for 
Movement (India), music from Matthias Duplessy, video from 
Christian Ziegler, interactive light installation controlled from 
Supercollider 
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Conclusion 


Chronotopia - a dance piece with the Attakkalari Centre for 
Movement (India), music from Matthias Duplessy, video from 
Christian Ziegler, interactive light installation controlled from 
Supercollider 

premiere at the Attakkalari India Dance Biennial in Bangalore (2 
shows) and 2 other shows in India (Chennai, Heggodu) 
tour in March in Germany (Kassel, Frankfurt) and Sweden (Vara) 

(video, performance Feb. 2009, in Bangalore, India) 


Hardware: light matrix and handheld lights - camera based 
videotracking 

■ 6 control boards for 6 CCFL lights based on XBee-Arduino 

■ 3 handheld CCFL lights with SenseStage MiniBee 
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Chronotopia - a dance piece with the Attakkalari Centre for 
Movement (India), music from Matthias Duplessy, video from 
Christian Ziegler, interactive light installation controlled from 
Supercollider 

premiere at the Attakkalari India Dance Biennial in Bangalore (2 
shows) and 2 other shows in India (Chennai, Heggodu) 
tour in March in Germany (Kassel, Frankfurt) and Sweden (Vara) 

(video, performance Feb. 2009, in Bangalore, India) 


Shared data: camera motion tracking, beat tracking, pitch 
tracking, light output data, frame timing 

■ 2 clients 

■ 15 datanodes 

■ with each 1 to 12 slots 
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JND/Semblance - a one-person installation piece by Chris 
Salter, Marije Baalman and Harry Smoak, interactive sound, 
light and vibration controlled from Supercollider 


public preview at Empac, Troy, NY, March 3-7, 2010 
ISEA, Essen, August 2010; Today’s Art, The Hague, September 2010 
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JND/Semblance - a one-person installation piece by Chris 
Salter, Marije Baalman and Harry Smoak, interactive sound, 
light and vibration controlled from Supercollider 

public preview at Empac, Troy, NY, March 3-7, 2010 
ISEA, Essen, August 2010; Today’s Art, The Hague, September 2010 

Sensing: 

■ 24 pressure sensing pads made of paper 

■ 3 SenseStage MiniBees 
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JND/Semblance - a one-person installation piece by Chris 
Salter, Marije Baalman and Harry Smoak, interactive sound, 
light and vibration controlled from Supercollider 

public preview at Empac, Troy, NY, March 3-7, 2010 
ISEA, Essen, August 2010; Today’s Art, The Hague, September 2010 

Sensing: 

■ 24 pressure sensing pads made of paper 

■ 3 SenseStage MiniBees 

Shared data: light settings, paper sensing, derived data, 
amplitude tracking 

■ 2 clients 

■ 17 datanodes 

■ with each 1 to 24 slots 
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Conclusion 


SenseStage workshop - one-week workshop at Concordia, 
using 25+ wireless sensor/actuation nodes and camera 
tracking, in which five projects were created by the participants. 

(video, workshop presentations May 2009, Montreal, Canada) 

http://sensestage.hexagram.ca/workshop/ 
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using 25+ wireless sensor/actuation nodes and camera 
tracking, in which five projects were created by the participants. 

(video, workshop presentations May 2009, Montreal, Canada) 

http://sensestage.hexagram.ca/workshop/ 

■ up to 10 clients 

■ up to 80 datanodes 

■ with each 2 to 12 slots 
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SenseStage workshop - one-week workshop at Concordia, 
using 25+ wireless sensor/actuation nodes and camera 
tracking, in which five projects were created by the participants. 

(video, workshop presentations May 2009, Montreal, Canada) 

http://sensestage.hexagram.ca/workshop/ 

■ up to 10 clients 

■ up to 80 datanodes 

■ with each 2 to 12 slots 

■ server was running for ca. 48 hours nonstop, with clients 
connecting and disconnecting, hardware nodes turning on 
and off 
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SenseStage MiniBee 

■ Board revision 

■ Extensive documentation 

■ Wireless bootloading 

■ Hopefully MiniBee for sale from second half of 2010 
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SenseStage MiniBee 

■ Board revision 

■ Extensive documentation 

■ Wireless bootloading 

■ Hopefully MiniBee for sale from second half of 2010 

SenseWorld DataNetwork 

■ Addition of enabling requests for “derived” data, such as 
statistical analysis, smoothed data, etc. 

■ Further integration with the Digital Orchestra Toolset 
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Many data streams require methods and tools to deal with this 
real time data to make sense of the information and to create 
meaningful content with it. 

■ organisation and visualisation of data 

■ feature extraction (environmental/gestures) 

■ fusion of data streams 

■ correlations between data streams 

■ emergent dynamics 


Development of tools integrated with the current infrastructure, 

as well as a lexicon of techniques. 
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http://sensestage.hexagram.ca 

Software available under the GNU/(L)GPL 
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